Comparison of neural architectures for sensor fusion

For technical speech recognition systems as well as for humans it has been shown that the combination of acoustic and optic information can enhance speech recognition performance. But it still remains an open question, at which stage of processing the two information channels should be combined. In this paper we systematically investigate this problem by means of a neural speech recognition system applied to monosyllabic words. Di erent fusion architectures of multilayer perceptrons are compared both for noiseless and noisy acoustic data. Furthermore, di erent modularized neural architectures are examined for the acoustic channel alone. The results corroborate the idea of separate processing of the two channels until the nal stage of classi cation.